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[57] ABSTRACT 

Plural efferent signal paths paired with plural conven- 
tional afferent signal paths respectively are provided 
between lower order cell-layers and higher order cell- 
layers of a neural network model. Once an output re- 
sponse has been derived from the higher order cell- 
layer, an efferent signal is transmitted through the effer- 
ent signal path paired with the afferent signal path con- 
cerned in the output response. Under the control of 
which efferent signal, the afferent signal path contribut- 
ing to the output response of the higher order cell-layer 
is affected by an excitatory effect, while the afferent 
signal path not contributing to the same is affected by an 
inhibitory effect. Hence the information processing 
consisting of both the associative memory and the pat- 
tern recognition provided with the faculty of segmenta- 
tion can be attained despite deformation and positional 
error of the input pattern. 

8 Gaims, 11 Drawing Sheets 
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FIG. 5 
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FIG. 8 
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HIERACHICAL INFORMATION PROCESSING 
SYSTEM 

The present application is a continuation-in-part ap- S 
plication of U.S. patent application Ser. No. 07/075,840 
filed on June 26, 1987 and now abandoned. 

TECHNICAL FIELD 

The present invention relates to a hierarchical infor- 10 
mation processing system for processing information in 
a cascade of successively arranged plural information 
processing layers in each of which plural nonlinear 
elements are two-dimensionally arranged, particularly, 
provided with an autoassociative recall type high grade 15 
associative memory function by which a pattern con- 
sisting of characters and figures excessively distorted in 
shape or excessively shifted in position, or accompanied 
with expansion or reduction can be readily and accu- 
rately recognized, and noise contained therein can be 20 
removed or defected portions thereof can be interpo- 
lated. 

TECHNICAL BACKGROUND 

When we watch a composite figure consisting of two 25 
patterns or more, we can segment it into individual 
patterns, and recognize each pattern separately. Even if 
one of the patterns to which we are paying attention is 
affected by noise or defects, we can recall the complete ^ 
pattern from which the noise has been eliminated and 
the defects restored. It is not necessary for perfect recall 
that the stimulus pattern is identical in shape to the 
pattern which we learned. Even though the pattern is 
distorted in shape or changed in size, we can recognize 35 
it and restore defects by interpolation. During the pro- 
cess of interpolation, we make full use of even slight 
traces in the defective parts of the pattern to which we 
are selectively paying attention, and recall the perfect 
original pattern. A hierarchical neural network model ^ 
which performs this function of the human brain is 
proposed in the prior art. 

As for a network required for recalling or recogniz- 
ing a complete pattern on the basis of an incomplete 
pattern, various kinds of autoassociative recall type 45 
associative memory networks have been conventionally 
proposed. However, most of these conventional asso- 
ciative memory networks do not work well unless the 
stimulus pattern is identical in size, shape and even 
position to a training pattern. That is, it could be satis- 50 
factorily operated only when an input pattern did not 
only consist of a previously learned training pattern, but 
also the former completely coincided with the latter in 
size, shape and position. Although an associative mem- 
ory system such as those employing an autocorrelation 55 
function accepting only the shift in position of the input 
pattern has been conventionally proposed, this conven- 
tional system also was completely impotent with regard 
to changes in size and distortion in shape of the pattern. 

The present applicant has already disclosed a pattern 60 
recognition system called "neocognitron" in the specifi- 
cations of Japanese Patent No. 1221756 (Japanese Pa- 
tent Application Publication No. 58(1983>53790) "A 
pattern recognition system" and Japanese Patent No. 
1279063 (Japanese Patent Application Publication No. 65 
60(1985)-712) "A pattern recognition equipment", 
which system is provided with an ability for correctly 
recognizing a pattern despite the effects mentioned 



above, that is, shifts in position, distortion in shape, and 
change in size and the like of the input pattern. 

However, these disclosed systems and equipment 
relate to no more than pattern recognition restricted 
within the above-mentioned version, and hence have 
not yet been provided with the ability to derive infor- 
mation regarding a phenomenon from incomplete or 
ambiguous informations with regard thereto by autoas- 
sociative recall, that is, the associative memory function 
required for attaining further higher grade pattern rec- 
ognition. 

On the other hand, the present applicant has also 
disclosed, as a system provided with the aforesaid asso- 
ciative memory function, a hierarchical information 
processing network in Japanese Patent Application 
Laid-open Publication No. 59-163679 "A hierarchical 
information processing network", which network is 
provided with the ability to successively select similar 
patterns resembling a previously learned training pat- 
tern in turn from plural input patterns, or, .to recall a 
complete pattern from an incomplete input pattern or 
an input pattern obscured by noise. However, this hier- 
archical information processing network has not yet 
been provided with the aforesaid ability for processing 
shifts in position and deformations in shape of the input 
pattern, similarly as the conventional associative mem- 
ory systems which have been proposed. 

DISCLOSURE OF THE INVENTION 

An object of the present invention is to obviate the 
above-mentioned various shortcomings, and to provide 
a novel hierarchical information processing system in 
which the above-disclosed hierarchical information 
processing network is offered with an ability such that 
pattern recognition can be achieved despite the affect of 
considerable defects such as deformation in shape, 
change in size and shift in position of the input pattern, 
as is different from the aforesaid "neocognitron", but 
with a higher grade autoassociative recall type associa- 
tive memory function and further with a segmentation 
faculty. 

Accordingly, in the hierarchical information process- 
ing system of the present invention, for achieving the 
information processing mainly through the aforesaid 
hierarchical information processing network: 

both of afferent signal paths extended upwards from 
lower order stages on the pattern input side to higher 
order stages on the recognition output side and efferent 
signal paths extended downwards from higher order 
stages to lower order stages on the contrary are pro- 
vided; 

input information supplied to the lower order stages 
is successively transmitted to the higher order stages 
through the afferent signal paths, as being successively 
processed in each of plural cell-layers of successive 
order stages, meanwhile, in the direction opposite 
thereto, an output derived from the higher order stages 
is fed-back to the lower order stages through the effer- 
ent signal paths; 

once an output response has been derived from the 
higher order stages, according to the action of the effer- 
ent signal transmitted from the higher order stages to 
the lower order stages, the afferent signal paths contrib- 
ute to cause the output response in the higher order 
stages and the information processing processes belong- 
ing thereto are affected by an excitatory effect, mean- 
while the remaining efferent signal paths and the infor- 
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mation processing processes belonging thereto are af- 
fected by an inhibitory effect; 

the ability of the pattern segmentation, that is, the 
distinction of components of a specified pattern from 
the remainder is offered by selectively extracting only 5 
the informations relevant to the specified pattern from 
plural patterns supplied to the lower order stages on the 
input side; or 

a complete pattern can be associatively recalled from 
an incomplete pattern by removing noise contained 10 
therein or by interpolating missing portions thereof. 

In other words, the hierarchical information process- 
ing system according to the present invention is fea- 
tured in that, for successively processing over a plural- 
ity of cell-layers each inter-layers of which a plurality of 15 
afferent signal paths for transmitting information signals 
are provided respectively, a plurality of efferent signal 
paths extended from higher order cell-layers to lower 
order cell-layers are respectively provided each inter- 
layers of the plurality of cell-layers, so as to make pairs 
respectively with the plurality of afferent signal paths 
extended from lower order cell-layers to higher order 
cell -layers, once an output response has been derived 
from the higher order cell-layer belonging to the affer- 
ent signal path, an efferent signal being transmitted 
through the efferent signal path making the pair with 
the specified afferent signal path transmitting an affer- 
ent signal which corresponds to the input response, the 
transmission of the information signal through each of 
the plurality of afferent signal paths being controlled 
respectively in response to the efferent signal, and 
hence, as to the causation of the input response derived 
from the higher order cell-layer under the control in 
response to the efferent signal, contributive one of the 
plurality of afferent signal paths being affected by an 
excitatory effect, while not-contributive one thereof 
being affected by an inhibitory effect. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a block diagram showing an example of 
hierarchical structure of a neural network model em- 
ployed for a hierarchical information processing net- 
work of the system according to the present invention; 

FIG. 2 is a block diagram showing an example of 
spatial connection between network elements in a part 
of the hierarchical structure as shown in FIG. 1; 

FIGS. 3 to 6 are block diagrams showing various 
kinds of embodiments of Weber-Fechner type circuit 
arrangement respectively; 

FIGS. 7 to 14 are block diagrams showing various 
kinds of embodiments of constituent cells of the neural 
circuit model as shown in FIG. 1; 

FIG. 15 shows five training patterns used for the 
self-organization of the network; 

FIG. 16 shows an example of the response of the 
network to juxtaposed patterns; 

FIG. 17 shows an example of the response of the 
network to an incomplete distorted pattern; and 

FIG. 18 shows an example of the response of the 
network to superposed patterns. 

1, 4 . . . gain-controlling adjustable resistor 

2, 8 . . . operational amplifier 
3 . . . voltage-control type adjustable resistor 
5, 7, 9, 10 . . . resistor 
6 . . . standard voltage source 
11 . . . half- wave rectifier 
14 . . . multiplying element 
12, 13, 15, 16, 17 . . . linear summation element 
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18 . . . dividing element 

19 . . . shunt type performance element 

21, 23, 54, 55 . . . adjustable gain element 

22, 37, 40, 47, 52, 58, 59 . . . total sum adder 

24, 30, 42 . . . adjustable gain amplifier 

25, 48, 60 . . . subtractor 

26 . . . shunt type divider 

27 . . . learning algorism 
28, 41, 66 . . . gain controller 
29 . . . adder 

31, 49, 61 . . . half- wave rectifier 
36 . . . square type nonlinear amplifier 
38 . . , square root type nonlinear amplifier 
39, 46, 51 . . . fixed gain amplifier 
53 . . . minimum output circuit 
64, 70 . . . maximum output circuit 
56, 57, 63 . . . buffer amplifier 
62 . . . phototransistor 
67 . . . saturation amplifier 
20 68, 69 . . . comparator 
71 . . . AND gate. 

BEST MODE FOR CARRYING OUT THE 
PRESENT INVENTION 

25 Generally speaking, when a composite pattern con- 
sisting of a combination of more than two patterns is 
presented, a human being is able of observing one of 
these patterns at a time with attention in order and as a 
result, to individually recognize what is each of these 
30 patterns, as well as to effect the aforesaid pattern seg- 
mentation. Besides, at the same time, even when the 
pattern observed with attention is deteriorated by de- 
fects or noise, a human being is able to associate a com- 
plete pattern by interpolating those defects or by re- 
35 moving the noise. In this connection, it is not required 
that the presented pattern completely coincides with 
the previously learned training pattern in shape or in 
size. A human being is provided with an ability for 
completing the original, even when the shape thereof is 
40 somewhat distorted or the size thereof is changed, by 
directly interpolating defects of the distorted pattern or 
by directly removing noise therefrom, and further by 
effectively utilizing a slight trace remaining in the de- 
fective portion of the attended pattern. The information 
45 processing system of the present invention has been 
proposed on the basis of the neural network model 
provided for realizing this ability of a human being. 

A preferred embodiment of the present invention will 
be described in detail hereinafter by referring to the 
50 accompanying drawings. 

An embodiment based on a multilayered network 
consisting of network elements arranged in layer state 
will be described hereinafter. However, as for the con- 
crete realization, it is enough to carry out the same 
55 algorithm, so that it is not necessary at all to restrict the 
embodiment to be described within the structure of the 
above multilayered network. 

In general, the neural network model used for the 
pattern recognition equipment is composed of a combi- 
60 nation of plural cell-layers in which plural cells are 
arranged in multilayer state, and hence the hierarchical 
structure as shown in a block diagram of FIG. 1 is ex- 
emplified as of four layers. In this block diagram, a 
circular mark o indicates a cell. Although each of the 
65 cell layers actually contains plural cells, plural cells of 
each kind are represented by only one cell in FIG. 1. 
Connections as indicated by single or double lines in 
FIG. 1 exist between these cells, the single line showing 
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the existence of one-to-one connection between hetero- 
geneous cells corresponding to each other, while the 
double line showing the existence of converging or 
diverging connection between two groups of cells. 

In this regard, the afferent signal is processed in the 5 
same structure as that of "neocognitron" as described in 
the specifications of Japanese Patent Nos. 1221756 and 
1279063. In this structure, a feature-extracting cell u s 
(including Uji, u f 2» Uj3 and so forth) extracts the feature 
of the pattern in association with an inhibitory cell u Jv . 10 
Hence, only when a specified feature exists in a speci- 
fied portion of the input cell-layer, an output response is 
derived therefrom. On the other hand, a feature-inte- 
grating cell u c extracts the same feature also. However, 
this cell u c receives output responses respectively de- 15 
rived from plural feature-extracting cells u, positioned 
in different portions of the input cell layers, so that, 
even when the position for indicating the same feature 
in the input layer is shifted to some extent, the feature- 
integrating cell u c itself continues to extract the same 20 
feature. An aspect of spatial interconnections between 
these cells u 5 and u c is shown in FIG. 2. 

More specifically, all the cells employed in the neo- 
cognitron are of the analog type: i.e., the input and 
output signals of the cells take non-negative analog 25 
values. Each cell has characteristics analogous to a 
biological neuron, if we consider that the output signal 
of the cell corresponds to the instantaneous firing fre- 
quency of the actual biological neuron. 

In the afferent signal path neocognitron, we use two 30 
different kinds of cells, i.e., S-cells and C-cells. As a 
typical example of these cells, we will first discuss the 
characteristics of an S-cell. 

An S-cell has a plurality of input terminals, either 
excitatory of inhibitory. If the cell receives signals from 35 
excitatory input terminals, the output of the cell will 
increase. On the other hand, a signal from an inhibitory 
input terminal will suppress the output. Each input 
terminal has its own interconnecting coefficient whose 
value is positive. Although the cell has only one output 40 
terminal, it can send signals to a number of input termi- 
nals of other cells. An S-cell has an inhibitory input 
which causes a shunting effect. 

S-cells and C-cells are excitatory cells, i.e., the output 
terminals of those cells are connected only to excitatory 45 
input terminals of other cells. The relation between 
excitatory inputs and inhibitory inputs will be described 
more completely by showing mathematical formulas. 

As described above, the afferent signal gradually 
integrates local features into a global feature, as the 50 
extraction and the integration of features are repeated in 
each layer of the multilayered network, and hence, the 
feature-integrating cell u c in the final layer globally 
observes the feature extracted from the whole input 
layer, namely, the input pattern, and, as a result, what is 55 
the input pattern can be recognized. 

In other words, an output response of the final cell- 
layer is derived from only one feature-integrating cell 
u c corresponding to the category of the input pattern in 
response to what is the category concerned. In these 60 
processes of the extraction and the integration of fea- 
tures, the shift in position of these features from each 
other is gradually accepted, and hence, accurate pattern 
recognition can be realized in the final cell-layer with- 
out any affection of shift in position as well as distortion 65 
in shape of the pattern supplied to the input cell-layer. 

As for the above mentioned processing of the afferent 
signal, the same structure of network as that of "neo- 



cognitron'* as disclosed in the specification of Japanese 
Patent Nos. 1221756 and 1279063 can be employed. 

In contrast with the above, the efferent signal is trans- 
mitted from the higher order cell-layers to the lower 
order cell-Layers through a feature-indicating cell w, 
and an integrative feature-indicating cell w c which 
make pairs with the feature-extracting cell u s and the 
feature-integrating cell u c as described above, respec- 
tively. That is, the efferent signal is transmitted down- 
wards through the path just opposite to that of the 
afferent signal transmitted upwards. 

It is enough to attain the above situation that the 
coefficients of connections between each cells are ar- 
ranged such as the efferent signal derived downwards 
from the feature-indicating cell w 5 is transmitted 
through the path just opposite to that through which 
the afferent signal is transmitted toward the feature- 
extracting cell u, paired therewith. In this regard, a 
feature-indicating inhibitory cell w jy is an auxililary cell 
corresponding to an inhibitory cell Ujyfor inhibiting the 
afferent signal. As for the structure concerning these 
cells, the substantially same as that of the hierarchical 
information processing network as disclosed in Japa- 
nese Patent Application Laid-open Publication No. 
59-163679 can be employed. 

However, the path of the efferent signal transmitted 
downwards from the integrative feature-indicating cell 
w c to the feature-indicating cell w, in the preceding 
cell-layer cannot be individually set up for the conve- 
nience of the efferent signal concerned. It is because the 
feature-integrating cell u c paired with the integrative 
feature-indicating cell w c is arranged so as to derive an 
output response, whenever any one of plural feature- 
extracting cells Us in the preceding cell-layer derives an 
output therefrom. So that, the efferent signal diver- 
gently derived from the integrative feature-indicating 
cell w c is transmitted to plural feature-indicating cells 
Wj, these cells Wj further receiving a signal operating as 
a gating signal from the feature-extracting cell u s corre- 
sponding thereto and hence being arranged to derive an 
output only when the signals are derived from both of 
the feature-extracting cell u^and the integrative feature- 
indicating cell w c thereto. According to this arrange- 
ment, as for the efferent connection from the integrative 
feature-indicating cell w c to the feature-indicating cell 
Wj also, the efferent signal can be transmitted down- 
wards through the same path as the afferent signal 
paired therewith and transmitted upwards from the 
feature-extracting cell u, to the feature-integrating cell 
u c . 

On the other hand, the efferent signal is not only 
affected by the afferent signal as mentioned above, but 
also affect the afferent signal on the contrary. That is, 
the fact that an output is derived from one of the fea- 
ture-integrating cells u c in the highest order stage of the 
network means that the pattern belonging to the cate- 
gory corresponding to the feature-integrating cell Uc 
concerned has been recognized through the network 
concerned. In this situation, the efferent signal transmit- 
ted downwards from the feature-integrating cell u c in 
the highest order stage is arranged so as to be supplied 
only to the cells directly related to the recognition of 
the pattern recognized in this situation. In this regard, in 
a situation where the input pattern supplied to the low- 
est order input stage is a composite pattern consisting of 
plural patterns, outputs of the intermediate stage in the 
afferent signal path are not necessarily derived only 
from the cells corresponding to the feature of the pat- 
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tern recognized through the final stage, but also derived nents corresponding to the recognized pattern appear in 
from the cells corresponding to the features of the re- the cell-layer, in which the integrative feature-indicat- 
maining patterns. In order to leave only the outputs ing cell w co of the initial stage exists, in the just same 
derived from the cells corresponding to the feature of form as the deformed input pattern. Moreover, even if 
the finally recognized pattern and to erase the outputs 5 defected portions exist in this deformed input pattern, 
derived from the remaining cells, the feature-integrat- the interporation which naturally corresponds to the 
ing cell u c is provided with an operational effect similar deformed input pattern is applied onto those defected 
to the biological "habituation", so as to gradually lower portions, and hence a deformed but indefectible pattern 
the gain between the input and the output thereof with appears in the cell-layer in which the integrative fea- 
the lapse of time. However, in conjunction therewith, a 10 ture-indicating cell w w of the initial stage exists, 
signal for forcibly recovering the lowered gain is ar- In this regard, a feedback signal x derived from a 
ranged to be supplied to the feature-integrating cell u c maximum detector MD which is presented in the right 
from the integrative feature-indicating cell w c paired end portion of FIG. 1 is transmitted to all of the feature-, 
therewith. As a result, the feature-integrating cell u c extracting cells u Jt only when an output is not derived 
belonging to the afferent path, through which the effer- 15 from all of the feature-integrating cells u f of the final 
ent signal is transmitted downwards, is affected by an stage. This feedback signal x lowers the selectivity of 
excitatory effect, and hence the lowering of the gain is the response of the feature-extracting cell u s similarly as 
not caused. the aforesaid output of the threshold controlling cell 

The output of the integrative feature-indicating cell w„, and hence, even if the feature of the input pattern is 
w c does not only affect the feature-integrating cell u f 20 incomplete, the feature-extracting cell u s is arranged so 
upwards corresponding thereto with the excitatory as to extract the incomplete feature concerned. As a 
effect, but also affects the feature-extracting cell u s . In result, any one of the plural patterns supplied as the 
this regard, the fact that, although an efferent output is input is recognized in the first place, 
derived from an integrative feature-indicating cell w c , As described above, only a part of the input compos- 
any afferent output has not been derived from the fea- 25 ite pattern which corresponds to the specified one pat- 
ture-integrating cell u c corresponding thereto means tern of the plural patterns contained therein is separated 
that the feature of the pattern, which should be taken therefrom, so as to appear in the cell-layer in which the 
charge of by the feature-integrating cell u c concerned, integrative feature-indicating cell w«of the initial stage 
fails to be extracted thereby. In this situation where, exists, the category of the separated pattern concerned 
although the efferent output is derived from the integra- 30 can be determined, namely, recognized in response to 
tive feature-indicating cell w c , any afferent output is not which one of the plural feature-integrating cells u c of 
derived from the feature-integrating cell u c correspond- the final stage from which one output has been derived, 
ing thereto, this situation concerned is detected by a In short, the segmentation in the pattern recognition can 
threshold controlling cell w^. When an output is de- be effected. In this respect, if the output of the cell-layer 
rived from the threshold controlling cell Wc*as a result 35 in which the integrative feature-indicating cell w w of 
of this detection, this output is operated so as to weaken the initial stage exists is regarded as an autoassociative 
the affection of the inhibitory, namely, negative signal recall type associative output, an associative memory 
supplied to the feature-extracting cell u„ and hence the network which is never affected by the deformation and 
selectivity of the feature extraction effected by the fea- the positional error of the input pattern can be realized, 
ture-extracting cell u s concerned is weakened. In short, 40 In this situation where any one of the plural input 
the feature-extracting cell Uj supplied with the output of patterns has been recognized, for setting about the rec- 
the threshold controlling cell w C xis forced to respond to ognition of another pattern thereof, it is enough to stop 
the feature of the pattern which should be originally the flow of the efferent signal for an instant. The fea- 
extracted by the feature-extracting cell Uj concerned, ture-integrating cell u c is arranged such that, when the 
even if the feature is incomplete to some extent. In other 45 facilitating signal derived from the integrative feature- 
words, the feature-extracting cell Uj is forced to be oper- indicating cell w c corresponding thereto is stopped, the 
ated similarly as, when a human being observes a gain thereof lowered by the affection of habitation is 
blurred character, a human being extracts the compo- restored, meanwhile the gain thereof forcibly increased 
nents of the character concerned which should exist in by the affection of the facilitating signal is lowered in 
the blurred portions thereof with the help of the slight 50 the just same manner as by fatigue in response to the 
traces left in the blurred portions thereof. extent of the forcible increase of the gain. As a result, 

Once the feature has been extracted by the feature- after the above instant stop of the efferent signal, the 
extracting cell u*on the basis of at least left slight trace previously recognized components of the input corn- 
thereof as described above, the efferent signal is caused posite pattern has been made to hardly pass through the 
to be transmitted further downwards through the fea- 55 afferent signal path, and hence another pattern different 
ture-indicating cell Wj corresponding to the feature- from the initially recognized pattern is caused to be 
extracting cell u, concerned. recognized. Accordingly, it is possible that various 

Under the repetition of the above-mentioned opera- kinds of operations such as pattern recognition, segmen- 
tion, only the components of the pattern which is recog- tation, association and the like as described above is 
nized through the response of the feature-integrating 60 successively carried out one after another of the plural 
cell w c in the final stage is caused to appear in the cell- input patterns. 

layer in which the integrative feature-indicating cell With regard to the extent of the modifiable synaptic 
w^of the initial stage exists. Moreover, in this situation, connections as indicated in FIGS. 1 and 2, it is enough 
the efferent signal is caused to be transmitted down- to vary the extent of the afferent connection by means 
wards through the same path as the afferent signal cor- 65 of the autolearning as disclosed in the specificaiton of 
responding to the efferent signal concerned. So that, Japanese Patent No. 1279063, as well as to vary the 
even if the input pattern has been deformed from the extent of the efferent connection in association with the 
previously learned training pattern, only the compo- corresponding afferent connection by means of the 
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method similar as disclosed in Japanese Patent Applica- block diagram as shown in FIG. 1 of the hierarchical 

tion Laid-open Publication No. 59-163679. structure of the neural circuit model adopted for the 

In this regard, various kinds of marks respectively hierarchical information processing system according 

indicating various kinds of connections and synapses to the present invention will be concretely described 
together with the operations thereof are listed below. 5 hereinafter by referring to the hardwares thereof. 

= converging or diverging connection between two Before the concrete description of individual cells, a 

groups of cells -one-to-one connection between two circuit arrangement presenting the performance of 

cells corresponding to each other — ►unmodifiable excit- Weber-Fechner type which is used for the feature ex- 

atory (positive) synapse —modifiable excitatory (posi- traction which is one of important functions of the neu- 
tive) synapse —| unmodifiable inhibitory (negative) syn- to r al circuit model, and thereafter detailed structures of 

apse -□ modifiable inhibitory (negative) synapse individual cells will be described in order. 

-O heterosynaptic facilitating signal for controlling i n ^ s connection, the circuit arrangement of Weber- 

gain of target -o inhibition weakening signal for Fechner type has been disclosed in detail by the present 

controlling selectivity of target cell. inventor in Japanese Patent Laid-open Publication No. 

INDUSTRIAL APPLICABILITY 15 51 ^ 55 35 essentially referred hereinafter. 

(1) Circuit arrangement of Weber-Fechner type 

The segmentation in the pattern recognition is ex- First, distinctions of the input to output performance 
tremely difficult to be applied particularly as for the 0 f this circuit arrangement will be described by refer- 

. modified input pattern such as handwritten characters, ri n g to equations, 
and hence has been conventionally effected, for in- 20 This circuit arrangement consists of multi-input to 
stance, with the help of parallel crosses previously one-output type provided with positive namely excit- 

printed on manuscript papers. atory inputs and negative namely inhibitory inputs, 

In this regard, according to the present invention, as these inputs and one output of this circuit arrangement 

for any possibly deformed pattern, the segmentation can being furnished with respective analog values of zero or 

be correctly effected despite deformation. 25 positive namely not-negative value. 

On the other hand, with respect to the associative j n a situation where inputs uj, u 2 , . . . , ujyare supplied 

memory equipment, that of the conventional system has t o N positive input terminals respectively, while the 

been hardly prepared so as to be satisfactorily operated, ot her inputs vj, V2, . . . , vm are supplied to M negative 

even if positional error, deformation, expansion, reduc- mput terminals respectively, the output W of this circuit 

tion or the like is contained in the input pattern, mean- 30 arrangement is defined by the following equation, 
while that of the system according to the present inven- 
tion can be prepared so as to be correctly operated 
without the affection of the positional error, the defor- 
mation, the expansion, the reduction or the like. 

This situation can be realized by the fact that the 35 w ~ * 
information processing network employed for the hier- 
archical information system according to the present 

invention is newly provided with the efferent signal where 
path together with the conventional afferent path and 

that such an operation is prepared between the afferent 40 > ^ (2) 

and the efferent signals, so as to facilitate the flow of 4>(x) = { *** " * 

those signals corresponding to each other, as well as to \ w < ) 
attenuate the flow of those signals not corresponding to 

each other. In tms regard, a n and b m are gains of n-th positive 

As is apparent from the mentioned above, according 45 in P ut terminal and m-th negative input terminal respec- 

to the hierarchical information processing system of the tiv ^ th « val . u es thereof being generally increased as 

present invention, it is possible to realize a novel pattern tnc self-organization is progressed, 

recognition equipment provided with an extremely The circuit arrangement concerned presents perfor- 

higher grade faculty being quite different from the con- mances specified as follows, 

ventional one such that the segmentation is effected by 50 For simplicity, an excitatory whole input e and an 

separating only a portion corresponding to one speci- inhibitory whole input h are defined by the following 

fied pattern from plural input patterns and that the au- equations respectively, 
toassociative recall type associative memory is effected 

for interpolating defected portions of the input pattern. N (3) 
Besides, in this situation also, the similar faculties as of 55 * ~ „- 1 ° n ' Un 
"neocognitron" as disclosed in the specification of Japa- 
nese Patent Nos. 1221756 and 1279063 are naturally A= % b m -v m (4) 
maintained such as the correct pattern recognition can m=i m 
be effected without the affection of the distortion in 

shape, the shift in position, the expansion or the reduc- 60 According to these definitions, the equation (1) can 

tion of the input pattern. be expressed by the following equation. 



(l) 

N 

1 + I flfl'U» 

M 1 

1 + 2 b m • v„ 

171=1 



DETAILED DESCRIPTION OF THE 



STRUCTURES AND THE ( 1 + e \ f e h \ 

INTERCONNECTIONS OF VARIOUS KINDS OF 65 w = * [TTT ' 1 ) = * {TTT ) 



(5) 



CELLS 



The structures and the interconnections of various As it is apparent from this equation (5), the excitatory 
kinds of cells which are employed in an exemplified input is supplied in a form of total sum of values defined 
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by the equation (3), so as to take effect additively, while 
the inhibitory input takes effect in shunt as expressed by 
the term 



(-■**) 



i + h is derived from the operational amplifier 2, so as to be 

supplied to a second operational amplifier 8 through a 
of the equation (5). resistor 7. An output of this operational amplifier 8 is 

When the inhibitory whole input h is small, that is, fed back to the input side thereof through a resistor 9, to 
the condition 0^h< < 1 is satisfied, the equation (5) can 10 which side the standard voltage from the voltage source 
be approximately expressed as follows. 6 is su PP Iied al s° through a resistor 10. Thereby, the 

output 

W-6(e-h) (6) 

This expression is coincident to the performance of 15 f ] J t ' - i ^ 

the "analog threshold element", the faculty of which ^ 1 + ' 
has been conveniently recognized to be that of the 

model of neural system or the visual pattern processing is obtained from lhe second operational amplifier 8, 

circuit. 20 w ^' c ^ sat * s fi es tnc relation of the equation (5), In addi- 

On the other hand, in a situation where the excitatory tion ' this out P ut is *»PPlkd to a half-wave rectifier cir- 

whole input e and the inhibitory whole input h are ™* "'f* t0 ° bta [ n the OUt ? Ut . . 

increased in proportion to each other, that is, the condi- * 4 ^ws another example of the circuit arrange- 

tion e=«, h=,x is satisfied, the equation (1) is ex- ' "W" 8 ^Tl ^ eXemphfied CirCUlt 

oressed as follows 25 a ™ngement, the excitatory inputs ui, u 2 , .... m N are 

H supplied to respective positive input terminals of a lin- 
ear summation element 12 through gain-controlling 

w = (g ~ v)x < 7) resistors 1-1, 1-2, . . . , 1-N respectively, while the inhibi- 

1 + tory inputs vi, V2, . . . , vji/are supplied to respective 

3 0 positive input terminals of another linear summation 
where € and tj are constant and €>tj. This expression element 13 through gain-controlling resistors 4-1, 4-2, . 
corresponds to the input to output performance consist- . . , 4-M. The output h of this element 13 is supplied to 
ing of the logarithmic performance of Weber-Fechner a negative input terminal of the summation element 12 
type added with a saturation performance, and hence is and further to a multiplying element 14 also. This multi- 
coincident to that which is generally adopted as an 35 plying element 14 is supplied with an output r of the 
empirical formula for approximating the nonlinear input summation element 12 also, so as to derive a product hr 
to output performance of sense system in the physiol- of both inputs h and r, which product is supplied to a 
ogy and the psychology. Accordingly, a circuit pro- negative input terminal of the aforesaid summation ele- 
vided with the input to output performance similar to ment 12 - The output r as defined by the following equa- 
that of a live body can be realized by employing the 40 tion b derived f rom this summation element 12. 
circuit arrangement concerned as a constant of the pro- 
cessing circuit for visual information and aural informa- 
tion. 

Various structural examples of the aforesaid circuit 
arrangement provided with the input to output perfor- 
mance as defined by the equation (1) will be described 
hereinafter by referring to accompanied drawings. 

FIG. 3 shows an example of the circuit arrangement This reformed equation satisfies the relation of the 

employing adjustable resistors of voltage control type. 50 equation (5). This output r is supplied to the half- wave 

In this exemplified circuit arrangement the excitatory circuit 11 similarly as in the aforesaid exemplified ar- 

inputs uj, U2, . . . , u;v are supplied to an operational rangement, so as to obtain the output W. 

amplifier 2 in a form of total sum thereof obtained by Next > FIG - 5 shows still another example of the cir- 

common connection through gain-controlling adjust- cuit arrangement employing a dividing element. In this 

able resistors 1-1, 1-2, . . . , 1-N respectively. An output 5 * exemplified circuit arrangement, the excitatory inputs 

of this operational amplifier 2 is fed back to the input u >' u ?> 7 ■ • J 1 "*™ supplied to respective positive input 

terminal of the amplifier concerned through a voltage tenninab of still another linear summation element 15 

control type adjustable resistor 3. A controlling voltage Ar °" gh S a ' n -™ntrolhng adjustable resistors 1-1, 1-2, . . 

input terminal of this adjustable resistor 3 is supplied ' ' ^ "S^&T the f mhlbltoI > in P uts ^ V *V 
.1. * ♦ 1 « f • T... . . FF 60 . - , v^/are supplied to respective positive input termi- 

with the total .sum .h of inhibitory inputs v1.v2 .yj, nd$ of %m Iinc ^ sun ^ ation e £ ment 16 

which is obtained by common connection through gain- ihrQa ^ k g ain ^ on t r olling adjustable resistors 4-1, 4-2, . . 

controlling adjustable resistors 4-1, 4-2. ... f 4-M respec- , ; 4_ M rcspect ively ( so as to obtain a total sum h thereof. 

lively, so as to obtain an output corresponding to the This totaJ sum output h is supp ii ed t0 a nega tive input 

term 1/(1 -f-h). Besides, an input terminal of the opera- 65 terminal of the aforesaid summation element 15, so as to 

tional amplifier 2 is connected also with a voltage obtain the output (e-h), and further is supplied to first 

source 6 of a standard voltage (a voltage of unit 1) positive input terminal of still another linear summation 

through a resistor 5. Thereby, an output element 17 also, which element 17 has a second positive 



45 



e—h — hr=*r 

This equation can be reformed as follows. 
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input terminal, so as to obtain the output (1+h) by lower order of feature-integrating cell u^ an input ter- 

supplying the standard voltage (+ 1) to the second posi- minaJ corresponding to the feature-extracting inhibitory 

tive input terminal. In succession thereto, the respective cell Ujy, gain-adjustable elements 21-1, 21-2 21-N, 

outputs (e-h) and (1 -f-h) of the summation elements 15 an adder 22, a gain-adjustable element 23, gain-adjusta- 

and 17 are supplied to a dividing element 18, so as to 5 ble amplifier 24, a subtractor 25, a shunt type divider 26, 

obtain an output gain-adjustable amplifier 30, a half- wave rectifier 31, 

and output terminals corresponding to the higher order 

e - ft u c cells are just the same to those in the wellknown 

1 + * ' conventional circuit arrangements exemplified as 

10 shown in FIGS. 3 to 6 respectively. 

This output satisfies the relation of the equation (5) and In other wordS) the input terminals receiving signals 

hence is supplied to the half-wave rectifier circuit 11 from thc lowcr order of featurc mtegrating ce n s Uc 

similarly as in the above exemplified arrangement, so as correspond to the aforesaid excitatory input terminals 

to obtain .the output W Ul( U2f ujv respectively, and the gain-adjustable 

In addition, FIG. 6 shows further another example of 15 elements 21-1, 21-2, . . . f 21-N correspond to the gain- 

the circuit arrangement employing an element having contro n ing adjustable resistors 1-1, 1-2, . . . , 1-N respec- 

the shunt type performance. In this exemplified circuit tiveIy Qn tne other hanc ^ h ^ satisfactory that ^ 

arrangement, the process in which the output (e-h) is output of the g ain . ad j ustab i e element 23 is similar to 

derived from the linear summation element 15, while that of the linear summation e!ement 13 having thc 

the output h is derived from the linear summation ele- 20 inh i bitor y inputs . Bes i des , the learning algorism 27 is 

ment 16 is simdar to that tn the example aforesaid by pmMed for making the ncutral circuit modcI of tnc 

referring to FIG. 5. However in this example as shown , invention t0 sense the ified b tne 

m FIG. 6 these outputs (e-h) and h are supplied to an leami bcfore cffcctin ^ m re ition md 

element 19 having the shunt performance, so as to ob- discrimination of the visual pattern , s0 as t0 prev iously 

tain t e output 25 xt tbe mtens i tv 0 f t h e connection between each cells, 

that is, concretely speaking, so as to previously set the 

* ~ * , gain-adjustable elements. 

+ The significant difference of the structural example as 

j. » u * ■ • m * *v ♦ • *u shown in FIG. 7 from the conventional structure is the 

The process succeeding thereto is similar to that in the 30 ^ ? A . _ - ■ ^ r n 

e * 1 • i 4t _ „ t .„. . . . connection of the respective inputs from the v/cx cell 

aforesaid example and hence the output W is derived . . w _ . . *Y , - . . ' . 

e *u 1. ir *-r . 4 and the MD circuit with the gain controller 28. The 

from the half-wave rectifier circuit 11. . - 4l _ . , f r ... 

t 1. r *j 1 *v * 11 input from the threshold controlling cell Wcx is provided 

In each of the aforesaid examples, the gain-controll- . H fc ... 4 . - ... r * . . 

j. . . ,, t< xfj**fii»# u for controlling the threshold of the selectivity for fea- 

mg adjustable resistors 1-1 to 1-N and 4-1 to 4-M can be . 5 . A , .. , A . . . , 

a ~r+u^<. _ u «c w„ ture extraction in the Ujcell such as the selectively con- 
formed of those such as the resistivity is varied by dnv- 35 . . * A • * * iL n 

ing a conventional adjustable resistor in response to a cerned m res f°™. ,0 th V np ™ fro . m thc *f\ CeU ' wh '? e 

controlling signal or by any electronic means such as more r °« , lcss « teorted P a " en \ K "*f ded ." 

FET, and further can be replaced by fixed resistors in * e s P e< f le(1 Pa"em «> as to adjust the gain of this 

case the function of self-organization is not necessarily cir^t element for reducmg the output h 

required 40 other hand, the input from the MD circuit is 

As described above, the circuit arrangement present- the Redback signal x from the maximum detector MD 
ing a Weber-Fechner type logarithmic performance can « ^ n m nG ; w * ,ch f ?*"* «gna» " g^ner- 
be provided by the above several examples, and hence ated when any of u c3 cells in the last stage of this neural 
can be employed as the constituent of the self-organiz- cir ™ x model d ° es not * e r nerate Us own out ? ut * s ° as t0 
ing circuit in a benefitable situation where any fear that 45 the selectivity for feature extraction also, 
the output is infinitely increased as the input gain is circult elements effect the function such that 
increased, similarly as in the conventional circuit ar- the neural circuit model 15 made to previously learn 
rangement and further the composition of self-organiza- P lural specified patterns and hence, for instance, in a 
tion system is facilitated, that is, the readjustment of situation where, although any of those specified pat- 
gain is not necessary and hence the circuit arrangement 50 tcrns * P rcscntcd thcret0 ' a11 of the out P uts thereof are 
is simplified zero, the selection width of feature extraction, which is 

Moreover, the circuit presenting input-to-output per- regarded as narrow, is spread, 

fonnance similar to the sense system of a live body can ( 3 > Feature-extracting Inhibitory Cell u„ paired with 

be readily realized by the aforesaid circuit arrangement, the feature-extracting cell u s 

so that the aforesaid circuit arrangement is benefitable 55 TOs feature-extracting inhibitory cell Ujvis employed 

for processing informations afforded by visual sense, for deriving an inhibitory input signal to be supplied to 

aural sense, tactile sense and the like. the aforesaid feature-extracting cell u 5 from the lower 

Under the preliminary knowledge wellknown ass order u r cells, as exemplified as shown in FIG. 8. In this 

described above, each constituent cells of the neural exemplified circuit arrangement, the square type nonlin- 

circuit model as shown in FIG. 1 according to the pres- 60 *** amplifier 36, the adder 37 and the square root type 

ent invention will be concretely described hereinafter in amplifier 38 are arranged such as a root mean square 

order, . type result is obtained from outputs of the lower order 

(2) Feature Extracting Cell Uj u c cells. However, the obtained result is not restricted to 

An exemplified structure of the u s cell according to this type, and hence it can be experientially arranged 

the present invention is shown in FIG. 7. 65 such as the inhibitory signal more efficiently functions 

AlS is apparent from FIG. 7, each constituents of this according to the combination of the result of this type 

Uj cell circuit arrangement according to the present with the Ujr cell, 

invention, that is, input terminals corresponding to the (4) Feature Integrating Cell u f 
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FIG. 9 shows a structural example of this cell Uc- This 
cell Uc receives respective outputs of plural u s cells and, 
even if the position of pattern presentation on the input 
layer is somewhat shifted, this cell u c maintains its out- 
puts. Besides, the output concerned is controlled by the 5 
signal derived from the cell W c on the efferent signal 
path through the gain controller 41 and the gain-adjust- 
able amplifier 42. The fixed-gain elements 39-1, 39-2, . . 
. , 39-N are employed for receiving the signals derived 
from the u* cells within the fixed mutual relation. 10 

(5) Threshold Controlling Cell w CJC 

FIG. 10 shows a structural example of this cell w^. 
In this structural example, the input signal is derived 
from the cell u, through the fixed-gain elements 46 re- 
spectively and mutually added through the adder 47, 
the output of which is subtracted from the signal de- 
rived from the cell w c and the subtraction output con- 
cerned is fed-back to the cells u s through respective 
half-wave rectifiers 49* so as to effect the adjustable 20 
excitatory connections therebetween. 

Both of the cells Uj on each layers belonging to the 
afferent signal path and the cells w c on each layers be- 
longing to the efferent signal path in the circuit arrange- 
ment as shown in FIG. 1 are simultaneously operated in 25 
the normal operation of this neural circuit model such as 
both of afferent and efferent signals or zero signals are 
output respectively. 

In a situation where the aforesaid balanced operation 
is disturbed, that is, for instance, although the efferent 30 
signal is derived from the cell w c , the afferent signal is 
not derived from the cell u* this cell w C x is activated 
such as the output thereof excitatorily affects the celt Uj, 
so as to forcibly derive the output from the cell u, con- 
cerned. In other words, the cell Wc* moderates the 35 
threshold of selectivity in the cell Uj, so as to facilitate 
the output of the cell u s through the afferent signal path. 

(6) Feature Integrating Cells w 4 

FIG. 11 shows a structural example of this cell w* 
belonging to the efferent signal path in the neural circuit 40 
model. In this connection, all of kinds of cells described 
hereinafter as shown in FIGS. 11 to 14 belong to the 
efferent signal path. 

This cell w 5 receives the signals derived from the 
higher order w c cells and the signal derived from the u s 45 
cell on the same layer. Those input signals from the 
higher order w c are supplied to an adder 52 through 
fixed-gain elements 51. The output signal of the adder 
52 is compared with the input signal supplied from the 
same order u* cell in a minimum output circuit 53, so as 50 
to selectively output the smaller signal among those 
compared signals. This output signal is efferently trans- 
mitted to the lower order w c cells and the Wjv cell suc- 
cessively through gain-adjustable elements 54-1, 54-2, . 
. . , 54-N and 55 and thereafter buffer amplifiers 56-1, 55 
56-2, . . . , 56-N and 57 respectively. 

In the aforesaid circuit arrangement, the minimum 
output circuit 53 is provided for selectively gating the 
efferent signals from the higher order w f cells in re- 
sponse to the signal derived from the same order Uj 
signal, so as to control the signal transmission through 
the efferent signal path in this neural circuit model. 

(7) Feature Integrating Inhibitory Cell w JV paired 
with the feature indicating cell w f ^ 

This cell w Jv is connected with the feature integrating 
cell W| in pair and hence the output thereof is transmit- 
ted to the plural lower order w c cells through efferent 



connections respectively. For simplicity, a drawing for 
exhibiting the structural example thereof is omitted. 

(8) Integrative Feature Indicating Cell w r 

FIG. 12 shows a structural example of this cell w Cl in 
which adders 58, 59, a subtractor 60 and a half-wave 
rectifier 61 are simply arranged as shown in the draw- 
ing, so that the detailed description of the operation 
thereof is omitted. 

(9) Photo-Electric Converting Cell u c0 

FIG. 13 shows a structural example of this cell in 
which a phototransistor 62, a buffer amplifier 63, a 
maximum output circuit 64, a gain-adjustable amplifier 
65 and a gain controller 66 are arranged such as the 
initial signal on the afferent signal path is adjustably 
excitatorily controlled by the final cell w« on the effer- 
ent signal path. In this connection, it is conventional 
that plural u f0 cells are arranged in matrix on the same 
plane. 

(10) Maximum Detector MD 

Maximum detector circuit MD is provided for detect- 
ing the u c 3 cell generating the maximum output among 
plural Uc3 cells, so as to derive the output "1" from the 
AND gate 71 corresponding to the 1^3 cell concerned, 
in the situation where the maximum output u C 3 is de- 
tected. On the contrary, in the situation where the maxi- 
mum output u C 3 cell is not detected, that is, all of the 
output of those plural u c a cells are zero, all of outputs of 
plural AND gates 71 are zero also, and hence the signal 
x is output therefrom as the efferent signal. 

FIG. 14 shows a structural example of this circuit 
MD, in which plural amplifiers 67 individually consist 
of saturation amplifiers or threshold elements for gener- 
ating the output *T* when the output is positive, while 
for generating the output "0" when the input is zero or 
negative. In this connection, plural comparators 68 and 
69 respectively present the performance such that, 
when the positive input is equal to or larger than the 
negative input, the output "1" is derived therefrom. 

THE STRUCTURE AND BEHAVIOR OF THE 
NETWORK 

(1) An outline of the Structure of the Network 

The model is a multilayered network consisting of a 
cascade of many layers of neural cells. Like the neocog- 
nitron or the hierarchical associative-memory model, 
the cells are of analog type; that is, their inputs and 
output take non-negative analog values corresponding 
to the instantaneous firing-frequencies of biological 
neurons. There exist both afferent and efferent connec- 
tions between cells in adjoining cell-layers. 

The heirarchical structure of the network in the pres- 
ent invention is illustrated in FIG. 1. Let the initial stage 
of the heirarchical network be called U ft the 1-th stage 
Uyand the highest stage Ul> Incidentally, FIG. 1 shows 
the case of L=3. The mark o in the figure represents a 
cell. Although cells of each kind actually exist in num- 
bers, only one cell is drawn in each stage. Between these 
cells, there are synaptic connections denoted by single 
lines and double lines in FIG. 1. A single line indicates 
that there are one-to-one connections between the two 
groups of cells, and a double line indicates that there are 
converging or diverging connections between them. A 
detailed diagram, illustrating spatial interconnections 
between neighboring cells, is shown in FIG. 2. 

At each stage, various kinds of cells, such as u* u Cl u sv , 
Wj, w f , Wjv, and w^, are severally arranged in two 
dimensional arrays. Notation U c /, for example, is used to 
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denote the layer of u r cells in the 1-th stage. In the fol- 
lowing equations, the output of u r cell in the 1-th stage, 
for example, is denoted by u/ (n, k), where n is two-di- 
mensional co-ordinates indicating the position of the 
cell's receptive-field center in the input layer U«,, and 
k(= 1, 2, . . . , K/) is a serial number indicating the type 
of feature which the cell extracts. Time t is assumed to 
take a discrete integer value. Incidentally, since we 
have Ko= 1 in stage U ft the output of a u^-cell, for ex- 
ample, might be denoted by u c </(n), in which k is abbre- 
viated. 

Just as in the neocognitron, the density of cells in a 
layer is designed to decrease with the order of the stage, 
because the cells in higher stages usually have larger 
receptive fields. Especially in the highest stage, only 
one urcell and w r cell exist for each value of k. Hence, 
expressions like ucz/(k), in which n is abbreviated, are 
sometimes used to denote the output of such cells. 

(2) Afferent Connections 



18 
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= \ 0 if 



0 

x < 0 



(10) 
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Parameter 07 is a positive constant determining the level 
at which saturation starts in the input-to-output charac- 
teristic of the cell. aXi/,*c,k)(£0) is the strength of the 
modifiable excitatory synapse coming afferently from 
cell uc/-i(n+v,K) in the preceding stage U/_i, and A/ 
denotes the summation range of v, that is, the size of the 
spatial spread of the afferent synapses to one Uj-cell. 
bXk)(^0) is the strength of the modifiable inhibitory 
synapse coming from inhibitory cell U/w(n). 

These modifiable synapses are reinforced by means of 
a similar algorithm to that used for learning-without-a- 
teacher in the neocognitron, and each Uj-cell comes to 
respond selectively to a particular feature of the stimu- 



has a structure and function similar to those of the neo 
cognitron. (Cells u* u ft and u JK correspond respectively 
to the S-, C-, and V-cells in the neocognitron.) 
After finishing the learning (or training) process. 



? f .^?!!^^^ 20 lus - In lhis specification, however, we shall consider 

only the state after learning, and assume that the synap- 
ses have already lost their modifiability. 
As can be seen from formula (9), the inhibitory input 
.._ B .carmng V or naming, process, to this cell acts in a shunting manner. The positive vari- 
feature-«tfact,ng cells u* with the aid of the feature 25 abler/(nk) wh j chwiUbe £ rlh er dhcusl J taterideler . 

S ^ « £r , ° m ^ mines the efficiencv ° f the inh *itory input to this cell, 

sumulus pattern (FIG. 2): that b. a u^ceU .s activated ^ mhjbi ^ whjch ^ an P inhibitorv sig . 

only when a particular feature is presented at a certain 1 . n • u * » i. • t_ . 

position in the input layer. The feature which the u,- na \ t0 th,S UrCeU Vf*" ^ ? * he We,ghted 

cells extract are automatically chosen by the model » root-mean-square of the s>gnals from the presynapt.c 

- - Urcells, mat is, 



itself during the process of learning in the same way as 
in the neocognitron. In lower stages, local features, 
such as a line of a particular orientation, are extracted. 
In higher stages, more global features, such as a part of 
a training pattern, are extracted. 

On the other hand, a urcell receives signals from a 
group of Ux-cells, all of which extract the same feature 
but from slightly different positions, and is activated if 
at least one of these u r cells is activated. Therefore, even 
if the feature is presented at a slightly different position, 
the u r cell usually still responds. 

Thus, in the afferent paths, the processes of feature 
extraction by the u r cells and the toleration of positional 
shift by the urcells are repeated, and local features are 
gradually integrated into more global features. Finally, 
the ucL-cells (i.e., the urcells in the highest stage, or the 
gnostic cells) integrate all the information in the stimu- 
lus pattern, and recognize it. In other words, in the 
highest stage, only one ucL-cell, corresponding to the 
category of the stimulus pattern, is activated. Since 
errors in the relative position of local features are toler- 
ated in this process of extraction and integration, the 
response of the ucL-cells in the highest stage is affected 
neither by a shift in position nor by a deformation in 
shape of the stimulus pattern. 

Mathematically, the output of a Uj-cell of stage U/is 
given by 



35 
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where a(y) represents the strength of the excitatory 
unmodifiable synapses, and is monotonically decreasing 
function of Ivj, which satisfies 



2 2 CX» = 
k— 1 vtAt 



(12) 



Incidentally, Ujrcells have the same characteristics as 
the root-mean-square cells introduced in the model 
"cognitron". 

The u r cells, which correspond to the C-cells in the 
neocognitron, are inserted in the network to tolerate 
positional errors in the features of the stimulus. The 
output of a urcell is given by 



2 dfr) 
vcDf 



"] 



(13) 



u l dn.k) ~ rftn.k) . 



where ] is a function specifying the characteristic of 
saturation, and is defined by 



W 60 



07 + 2 , 2 aKv,tc t k) . 



J a-\ (" + ?.*) 



07 + - 



r/(n.k) 



1 + 



rf(n.k) 



(14) 



where 



Parameter d/(v) denotes the strength of the excitatory 
65 unmodifiable synapses, and is a monotonically decreas- 
ing function of | v | , Hence, if at least one presynaptic 
urcell is activated in the area D/, to which these con-, 
nections spread, this u<-cell is also activated. 
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The variable g/(n,k) denotes the gain of the u^-cell, 
and its value is controlled by the signal from the wrcell 
in the efferent path. 

(3) Efferent Connections 

Efferent signals are transmitted to lower-order cells 5 
through w j- eel Is and w r cells, which make pairs with 
u r cells and u^-cells respectively. Here, the efferent 
signals are made to retrace the same paths as the afferent 
signals in the opposite direction. 

(3)-l The Efferent Connections from w r Cells to w r io 
Cells 

Let us consider the efferent signals from an arbitrary 
w r cells to the wrcells of the preceding stage (FIG. 2). 
Just as in the case of the hierarchical associative mem- 
ory model, it is assumed that there exists a mechanism to 15 
control the strength of the synapses in the following 
manner: after finishing the process of learning, the effer- 
ent connections descending from a w r cell are made to 
have a strength proportional to the afferent connections 
ascending to the uj-cell which makes a pair with the 20 
w r cell. 

Thus, the output of a wrcell is given by 



n^n.*) 



= * a/. \ 2 



(15) 



2 ai+\{v.k,K)' 
vcAt+\ 



wi/+l( w - v -*> - J ^+l(v)*"iw+i<n 



•)] 



30 



where a/ is a positive constant. 

As you may seen by comparing of this equation to 
formula (9), the efferent signals descending from the 35 
w r cell are transmitted in the opposite direction through 
the same paths as the afferent signals ascending to the 
uj-cell. The inhibitory efferent signals via the Wjv-cell, a 
subsidiary inhibitory cell corresponding to the Ujy-cell 
in the afferent paths, are also transmitted in the opposite 40 
direction through the same paths as the afferent signals 
ascending via the u jv -cell. 

The output of the subsidiary inhibitory cell Wjy is 
given by 

45 



1 + 71+1 k = { 



2, bi+i(K) • ^ +} {n.k) 



(16) 



That is, the efferent connection from w,/ + i- to w„/+i- 50 
cell is the same as the afferent connection from u*w+i- 
to Uj/+ i-cell, where only the direction of the signal flow 
is reversed. The only difference between the efferent 
and the afferent paths is that the wrcell in the efferent 
paths has a linear input-to-output characteristics, while 55 
the Ujv-cell in the afferent paths has a root-means-square 
characteristic. The parameter Ti°in formula (16) is the 
initial value of the variable T/(n,k) in (9), and will be 
discussed later in connection with formula (22). 

As a result of the network structure described above, 60 
if an excitatory afferent path is formed to a Urcell from 
a Uf-cell, an excitatory efferent path comes to be formed 
automatically from the corresponding wrcell to the 
corresponding wrcell. When the afferent inhibitory 
path via the inhibitory u Jir cell is stronger than the direct 65 
excitatory path from the u r cell, the efferent inhibitory 
path to the wrcell via the inhibitory wrcell becomes 
stronger than the direct excitatory path from the wr- 



cell, and the w r cell comes to receive an inhibitory 
overall effect from the w r cell. In short, depending on 
whether a u r cell is afferently receiving an overall excit- 
atory or inhibitory effect from a Uc-cell, the correspond- 
ing w r cell also receives efferently an overall excitatory 
or inhibitory effect from the corresponding w r cell. 
(3)-2 Efferent Connections from w r Cells to w^Cells 
Corresponding to the afferent connections which 
converge to a urcell from a number of u r cells, many 
efferent connections diverge from a wrcell towards 
wrcells (FIG. 2). It is not desirable, however, for all the 
w r cells which receive excitatory efferent signals from 
Wc-cells to be activated. The reason is as follows: to 
activate a u r cell, the activation of at least one presynap- 
tic u r cell is enough, and usually only a small number of 
presynaptic u r cells are actually activated. In order to 
elicit a similar response from the w r cells in the efferent 
paths, the network is synthesized in such a way that 
each w r cell receives not only excitatory efferent signals 
from wrcells but also a gate signal from the corre- 
sponding urcell; and it is activated only when it re- 
ceives a signal both from u r and wrcells. 
Thus, the output of a wrcell is given by 



25 



1 [u' s {n.k), 2 dM - 



(17) 



where min[ ] is a function which takes the smaller value 
of the two arguments. 

Because of this network architecture, in the efferent 
paths from wrcells to wrcells, the signals retrace the 
same route as the afferent signals from Uj-cells to ur- 
cells. 

(4) The Action of Efferent on Afferent Signals 
(4)- 1 Facilitation 

As described above, efferent signal flow is affected by 
afferent signals. Efferent signals, however, are affected 
not only by afferent signals, but also influence the affer- 
ent signals flow. Activation of a single ucz.-cell in the 
highest stage means that the network has recognized a 
pattern of a category which corresponds to the acti- 
vated ucL-cell. Efferent signals descending from the 
activated ucL-cell are now sent only to cells relevant to 
the recognition of the pattern now recognized. If the 
stimulus presented to the input layer Uco contains two 
patterns or more, in the intermediate stages of the affer- 
ent paths, not only cells corresponding to features of the 
pattern now recognized at the highest stage, but also 
cells corresponding to features of the other patterns are 
usually activated. In order to preserve only the output 
of cells relevant to the features of the finally-recognized 
pattern and to suppress the output of the other cells, 
each Uc-cell is assumed to have the characteristic of 
"habituation", by which the gain between the inputs 
and the output of the cell is gradually attenuated with 
the passage of time. At the same time, however, the 
u r cell receives a facilitating signal from corresponding 
w r cell. This facilitating signal forces the attenuated 
gain to recover. Consequently, u r cells in the paths in 
which efferent signals are flowing do not have their 
gain attenuated because of the facilitation effect. 

Quantitatively, the gain of a urcell in the afferent 
paths [see (13)] is given by 

*/(«.*) = yi- g) _ W) + (l - yt) • *[y/w'a x (t>.k)] (18) 
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where y/ is a constant determining the speed of habita- 
tion of the Ur-cells, and lies in the range 0<y/= 1. The 
positive constant yf determines the signal level at 
which the saturation of facilitation begins. The gain at 
the initial state of the network is 



22 

associative memory 



model (Fukushima 



hierarchical 
1984). 

Quantitatively, the efficiency of the inhibition is de- 
termined by the variable y/in (9), and the value of y/is 
controlled by the neuromodulator x 5 as follows. 



gftn.k)=l 



(19) 



Incidentally, this effect of facilitation corresponds, 
for example, to the heterosynaptic facilitation observed 10 
in Aplysia (Castelluci and Kandel 1976). 

(4)-2 Controlling the Efficiency of the Inhibitory 
Input 

Consider a case in which a urcell is silent while the 
corresponding w^cell is activated by efferent signals. 15 
This means that the feature which the u^cell should 
extract has not been detected in the afferent paths. In 
other words, none of the u r cells presynpatic to this 
lie-cell could extract the feature which is supposed to 
exist there. It is a wcr-cell that detects the situation in 2 q 
which urcells do not yield an output. 

The output of a wcr-cell is given by: 



(22) 



1 + x' s /Ln,k) + x' xi 



(20) 



25 



In order to detect a condition in which a w^-cell, but not 
a u r cell, is activated, one might think that the subtrac- 
tion of the output of the Ur-cell from the output of the 3Q 
w r cell would be enough. However, this is not always 
the case, because the output of the Urcell might be 
attenuated by habituation. Hence, in the present model, 
instead of the output of the u<-cell itself, the signal con- 
verging from presynaptic u r cells is used (FIG. 2). 35 

To be more exact, the spreading area D/ of the affer- 
ent connection d/(v) to a wcr-cell from Urcells is deter- 
mined as a little wider than the spreading area D/of the 
afferent connections dXv) to a u^-cell. This is for the 
following reason: let a stimulus pattern be a slightly ^ 
deformed version of a learned pattern. The output of 
the Uc-cells elicited by the stimulus pattern itself might 
possibly arise at a slightly different position than that of 
the w r cells, which is determined by the response to the 
training pattern during the learning period. In order to 45 
prevent a spurious output from the wcrcells caused by 
a discrepancy between the output of these two groups 
of cells, the spreading area D/ is determined as a little 
larger than D/in formula (13). 

Now, if a wcr-cell is activated, it releases a kind of 5Q 
neuromodulator x s to corresponding Uj-cells. In this 
case, the neuromodulator is transmitted through paths 
inverse to the afferent connections converging to the 
u r cell from the u r cells. Quantitatively, the amount of 
neuromodulator Xs which acts on cell u^n,K) is 



(21) 



or neuromodulator is gradually accumulated with the 
passage of time, and, at the same time, decreases with an 
attenuation constant /J<0</?/^ 1). In the computer sim- 
ulation discussed later, we put £/=l. /?/' is another 
positive constant. 

The neuromodulator x x weaken the efficiency of the 
inhibitory signals from the u^-cell, and decreases the 
selectivity of feature extraction by the u r cell. Inciden- 
tally, a similar process has been hypothesized in the 
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where y/° is the initial value of y/. The variable x x f 
represents the amount of another neuromodulator re- 
leased by signal x from the maximum-detector MD, 
which will be discussed later [see (25)}. In the highest 
stage U/., no w CJ rcells exit, but formula (22) can be 
applied if Xj is assumed to be zero. 

Thus, neuromodulator from w C ;r cells makes Urcells 
respond even to incomplete features to which, in the 
normal state, no u r cell would respond. In other words, 
by control of the neuromodulator, u r cells search for 
even vague traces in defective parts of the stimulus 
pattern, and try to detect features which should exist 
there. 

Once a u r cell has been thus activated, the efferent 
signal now comes to be further transmitted to the lower 
stages via the Wrcell corresponding to the urcell. 
(5) Association and Segmentation (Initial Stage Uo) 
As described above, even if a number of stimulus 
patterns are simultaneously presented to the network, 
only components of the recognized pattern, determined 
by the response of ucL-cells in the highest stage, are fed 
back to layer Wco(i.e., the layer of w^cells in the initial 
stage). In this case, the efferent signals retrace the same 
route as the afferent ones. Hence, even if the stimulus 
pattern is a deformed version of the standard pattern 
which has already been learned, only the signal compo- 
nents corresponding to the recognized pattern appear in 
layer Wco for the identically deformed shape. Even if 
the stimulus pattern has some parts missing, these are 
interpolated smoothly, and a perfect pattern emerges in 
layer Wco. 

Therefore, the output of layer Wco can be inter- 
preted as an auto-associative recall from the associative 
memory. From a different point of view, the output of 
layer Wco can also be interpreted as the result of seg- 
mentation, where only components relevant to a single 
pattern are selected from the stimulus. The category of 
the segmented pattern can be judged (or recognized) by 
noting which ucz.-cell is activated. Thus, we can also 
conclude that segmentation and pattern recognition are 
simultaneously performed. 

In order to improve the ability to interpolate defects, 
the model in the present invention has such a structure 
that the recalled output of layer Wco is fed back posi- 
tively to layer Uco- That is, both the stimulus input p 
and the recalled output Wco are superposed in layer 
Uco» as shown in FIG. 1. Expressed mathematically: 

»CcKn)^gd{n)-max\^{n%wccKn)]\m (23) 



The grain g is given by formulae (18), (19) and (26) in 
the same manner as for the intermediate layers. Inciden- 
tally, stage Uo contains only u r and w^-cells, and the 
output of a Wf-cell is also given by formula (15). 
65 Since a positive feedback loop is included in the net- 
work, the recalled output, which is first generated only 
by the stimulus p, is fed back to the input, and another 
recalling process goes on with the newly-recalled out- 
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put as a new input. Therefore, even if the interpolation 
cannot be completed in a single recalling process be- (25) 
cause the defects are too large, the defects will gradu- 
ally be filled up during a chain of associative recall, and 5 x 'xi- 
a perfect pattern without any defect is finally obtained. 

As may be seen in formula (23), not only feedback 
signals form W C o but also the stimulus input p is still In other words, if all the ucz.-cells in the highest state 
given to layer Uco* even after the start of the chain of are silent, x* accumulates by a constant amount f$ x f- If at 
associative recall. This mechanism prevents unlimited 10 least one ucx-cells is activated, however, it decreases 
drift int he recall, and suppresses the appearance of with an attenuation ratio fi x f. 

useless patterns unrelated to the stimulus. A $ n *s already been shown in formula (22), 

(6) The Maximum-Detector neuromodulator x* due to signal x, together with 

When two or more patterns are simultaneously pres- f neuromodulator x, released by the wcr-cells, weakens 

ented, two or more u r cells may be activated simulta- 15 f he efficiency of mhibUion of feature-extracting cells u f 

neously, even in the highest stage U L . In such a case, the ° WerS I 1 ' i r ™™ $elecUvit * of 

. ' 4 ... * t it <• their response. As the result, feature-extracting cells u x 

stimulus pattern corresponding to the ucx-cell of max,- respond ^ even %Q imperfect featl f res or t0 

mum output is selected first, to start processing these stimuli mixed with irrelevant features, 
patterns in turn. This selection of the maximum-output 20 The Ionger the silent state of thc UCL ^ t]h continues, 
cell is done by the maximum-detector shown at far right the more is the accumulation of neuromodulator 
in FIG. 1. The maximum-detector selects the one which released by the maximum-detector. Hence, at least one 
is yielding the largest output from among the ucz.-cells. ucx-cell becomes activated after a certain time. 
The pattern of the category which corresponds to the The influence of the signal x on Uj-cells (or the value 
ucl-ccII thus selected can now be interpreted as recog- 25 offixi'm formula (25)) is not identical at all stages, but is 
nized. greater for u r cells in higher stages. This is useful be- 
Once the maximum-output u C f cell has been selected causc of Allowing. Let two or more patterns be 
in the highest stage, only the wci-cell which makes a simultaneously presented. In the higher stages, each 
pair of this ucL-cell is activated by the excitatory signal 30 has r a « reater ch ™ cc °[ h ™f Panted with 

from the maximum-detector, and the other wcx-cells "J 1 eva f f f* ur f to *f her W1 * the fe f ture wmch the 
_ . , „ , cell is due to extract because it has a larger receptive 

stay m a resting state Thus the output of a w^cell of fiejd Hencc thc selcctivity of rcsponsc sh * uld bc a P dap . 

stage U L ,s different from that of other stages, and is tively changed to extract the relevant feature without 
given by ^ bemg hindered by accompanying irrelevant features. In 

the lower stages, however, the control of selectivity is 
(2+) not so important, because each u^-cell has a smaller 
l if vW <*) = m«t«'c/<K)l > o receptive field, and has small chance of being presented 

^ with two or more features simultaneously. 

0 eke 40 Incidentally, during the period in which the x-signal 

is being sent from the maximum-detector, the gain g of 

tu^ r~ * *u 1 i re * * L i ^e u<r-cells in the network is assumed to be unchanged 

Therefore, in the network, only afferent paths rele- ^ M b eQm ^ 

vant to the pattern which is now being detected (or (7) Switching Atte ntion 

recognized) at the highest stage are facilitated by the 45 Supposc onc of the mu i t j p i y - P reser,ted stimulus pat- 
action of efferent signals from the activated wcx-cell. terns is 5eing attended to end recognized. In order to 
The mechanism of this facilitation has already been switch attention to another pattern, a momentary inter- 
discussed, ruption of the efferent signal-flow is enough. Each u r 
Sometimes all the ucL-cells in the highest stage are cell, after the disappearance of the facilitating signal 
silent. For instance, if too many patterns, are simulta- 50 from the corresponding Wc-cell, will have its gain low- 
neously presented, or if they are too close to one an- ered ^ tms has so far been kept high by the facilitating 
other, the signals from the different patterns hinder each si S nal - A decrease in gain occurs like fatigue, depending 
other, and are unable to elicit any response from the on the de S ree of the forced ^crease of gain until then. 

usjL-cells. Hence ucx-cells which receive signals from „ ° n other hand ' the u ^ ce11 wU1 recover its « ain ' if 

u i m . c i. • i this has so far been attenuated by "habituation". 

us/.-cells are also silent. Even when presenting a single * . . ' ««»^» 

, „ i • .» „ Quantitatively, the gam just after an interruption in 

stimulus pattern, there is also a chance that no ucL-cel *^*ce„^ ™ i n«. - *u r , /10 x 

.... . , t„ tn e efferent signal-flow is given, not by formula (18), 

will be activated if the stimulus pattern is greatly differ- but by 

ent in shape from the original pattern, or if it has large 

defects or a lot of noise. ^° 1 

The situation in which a] the ucx-cells are silent is */(«•*) » j + y/*gJ-'(nJk) 
detected by the maximum-detector, and the maximum- 
detector sends out signal x through the path shown in where y{ . is a positive constant 
FIG. 1. This x-signal is commonly distributed to all the 65 In the highest stage Vu the gain of the u^^lls is 
feature-extracting cells u, of all stages. usua i] y kept at 1 (i.e., y L = 1 in (18)). Just after the inter- 

The amount of neuromodulator x* released by this ruption of the efferent signal flow, however, the gin gi 

x-signal to a single u r cell is of the ucl-cc11 corresponding to the WQL-cell which has 
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been activated until the attention shift, is made zero to elicited is shown at t=4. In layer Ucl, only the gnostic 

prohibit a second selection of the same cell. cell for pattern "2" is now activated, because the gains 

Incidentally, the amount of neuromodulator x s and x x of the Uc-cells which have so far been responding 

is also assumed to be reset at zero just after the interrup- strongly to pattern "0" are now attenuated by fatigue, 

tion of the efferent signal flow. 5 Hence, also in layer W^ in the initial stage, only the 

Since the gain of the u r cells are thus controlled after pattern components of "2" appears, 

the interruption of the efferent signal flow, the signal Although patterns "0" and "2" contained in the stim- 

components of the pattern which has so far been recog- ulus are a little different in shape from the training pat- 

nized Finds it hard to flow through the afferent paths, terns (FIG. 15), the patterns fed back to layer W„ are 

and another pattern, different from the first one, will 10 identical in shape with the stimulus patterns now pres- 

usually be recognized. ented. 

Therefore the process of pattern-recognition, seg- FIG. 17 shows the response to a greatly deformed 

mentation and associative recall goes on for the multi- pattern which even has one part missing. Since the 

ply presented patterns in turn. difference between the stimulus and the training pattern 

(8) Computer Simulation 15 is too large, no response is elicited from layer mcl (not 

We will now study the behavior of the model in the . shown in FIG. 17), and accordingly, to feedback signal 

present invention by computer simulation. The scale of appears at layer W w at t=0. This situation is detected 

the simulated network is L=3. by the maximum-detector, and signal x is sent. At this 

We will discuss mainly the response of the network t=l, the gnostic cell for "2" becomes activated in the 

which has already finished learning, but we shall also 20 highest stage, because the threshold of the Uj-cells in the 

briefly describe the earlier process of learning. During network is decreased by the x-signal. In the pattern 

the learning period, five training patterns, "0", "1", "2", which is now fed back to layer W COp both ends of the 

"3", and "4" shown in FIG. 15, were repeatedly pres- missing part are already beginning to be interpolated, 

ented as a stimulus input p, and the self-organization of Partly interpolated, this signal, namely the output of. 

the network occurred in a similar manner as for the 25 layer W^ is given to layer Uco again, together with the 

neocognitron. In the initial state before learning, all the stimulus p. The interpolation continues gradually while 

modifiable synapses of the network were assumed to the signal is circulating in the feedback loop, and finally 

have a strength of zero. During the learning period, to the missing part of the stimulus is completely filled in. 

simplify the simulation all efferent signal-flow was As may be seen in FIG. 17, the missing part is interpo- 

stopped, and habituation of the u^-cells was assumed not 30 lated quite naturally, even though the difference in 

to occur. It is further assumed in the simulation that all shape between the stimulus and the training pattern is 

the modifiable synapses lose their "plasticity" after fin- considerable. Incidentally, in the pattern for which 

ishing learning. interpolation is already finished, the horizontal bar at 

FIGS. 16-18 show the behavior of the network the bottom of the "2" is shorter than in the training 
which has already finished learning. In these figures, the 35 pattern. But, however, short the horizontal bar is, the 
response of the cells in layers U OT and Wo> of the initial pattern is still a perfect character M 2". Hence, this corn- 
stage is shown in time sequence. The intensity of re- ponent of the pattern is left intact and is reproduced like 
sponse of the cells, which are arranged in a two-dimen- the stimulus pattern. Thus, the deformation of the stim- 
sional array, is shown by the density of the photo- ulus pattern is tolerated as it is, and only indispensable 
graphs. The patterns in the upper line show the re- 40 missing parts are naturally interpolated without any 
sponse of layer Uco, and the patterns in the lower line strain. 

show the response of layer W^ The numeral to the FIG. 18 shows an example of the response to a stimu- 

upper left of each pattern represents time t after the start lus consisting of superposed patterns. As may be seen in 

of stimulus presentation. The mark T to the right of the the figure, the pattern "4" is isolated first, the pattern 

numeral denotes that the efferent signal-flow has been 45 "2" next, and finally pattern "1" is extracted, 

momentarily interrupted just before this moment to (9) Effect of Invention 

switch attention. The stimulus pattern given to this The neural network model proposed in this invention 
network is identical to the response of layer Uco at t =0, has the ability of selectiv attention, pattern recognition 
which is shown in the upper left of each figure. (It and associative recall. When a composite stimulus con- 
should be noted that the input pattern p appears directly 50 sisting of two patterns or more is presented, the model 
in layer Uco at t— 0, because no response is elimited from in the present invention pays selective attention to each 
layer W w for t<0.) of the patterns one after the other, segments it from the 

FIG. 16 is the response to stimulus "20" shown in the rest, and regognizes it separately. From the engineering 

upper left. In layer Uct at the highest stage of the net- point of view, this invention could be effectively used as 

work, which is not shown in this figure, the gnostic cells 55 a design principle for a pattern recognizer with the 

corresponding to pattern "2" and "0" are both activated function of segmentation. If we want to extend the 

at t=0, but a larger response is elicited form the cell model, and to make a system which can read a docu- 

corresponding to "0". Hence, in layer Wei, only the ment including many different letters, it would be nec- 

cell corresponding to "0" is activated by the signal from essary to add more functions. For instance, it would be 

the maximum-detector MD. This signal x is fed back 60 useful to install a mechanism like eye-movement by 

further to layer W„, through the efferent paths. With a which the document is scanned by a small gazing-area 

lapse of time, habituation of the Uc-cells builds up gradu- covering only a few letters. In order to read a document 

ally in each layer. At t = 3, the components of pattern written from left to right in order, the network would 

"2", which is not being attended to, have already be- further need an external signal by which the signal paths 

come quite weak. 65 from the upper left part of the gazing-area were more 

At this time, in order to switch attention, the efferent facilitated than that from the lower right, 
signal-flow is interrupted for a moment, and the habitu- The model in the present invention can also be con- 
ation of the cells recovered. The new response thus sidered as a model of associative memory. In contrast to 
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earlier models, our model has the ability for perfect 
associative recall, even for deformed patterns, without 
being affected by their position. Hence, this invention 
can also be used effectively to repair imperfect patterns 
or to search for pictorial data from a large data-base. 

In this application, the macroscopic behavior of our 
model has been mainly discussed, and it has been shown 
to resemble that of the brain in many ways. Let us con* 
sider the characteristics of the receptive field of cells of 
the network. The cells in the afferent paths, especially 
in higher stages, have a large receptive field. If two 
stimuli are presented to the receptive field of a cell, only 
a stimulus which is being attended to elicit a large re- 
sponse, and the response to a stimulus which is not 
being attended to become greatly reduced. 

What is claimed is: 

1. A hierarchial information processing system for 
processing information in parallel over a plurality of 
cell layers comprising: 
a plurality of afferent signal paths extending from 
lower order cell-layers to higher order cell-layers 
for processing information signals, wherein said 
afferent signals are processed by a plurality of fea- 
ture-extracting cells Uj and feature-integrating cells 
u f alternatively and convergingly; 
a plurality of efferent information paths extending 
from higher order cell-layers to lower order cell- 
layers for transmitting an efferent signal, wherein 



to all feature-extracting cells u,so as to lessen the inhibi- 
tion to each target cell u s . 

3. A hierarchical information processing system for 
processing information in parallel over a plurality of 

5 cell layers as defined in claim 1, further comprising an 
inhibiting cell Ujv convergingly receiving a pattern sig- 
nal from the said feature-integrating cell u c of the pre- 
ceding stage and sending output signals via modifiable 
inhibitory connections to the said feature-extracting cell 

10 u* 

4. A hierarchical information processing system for 
processing information in parallel over a plurality of 
cell layers as defined in claim 1, further comprising a 
feature-indicating inhibitory cell w Jf connected to the 

15 feature-indicating cell w y and connected divergingly 
and unmodifiably to the integrative feature-indicating 
cell w c in the lower stage. 

5. A hierarchical information processing system for 
processing information in parallel over a plurality of 

20 cell layers as defined in claim 1, wherein the afferent 
signal path of serial connection of the feature-integrat- 
ing cell u c and the feature-extracting cell Uj and the 
efferent signal path of serial connection of the integra- 
tive feature-indicating cell w f and the feature-indicating 
25 cell Wj are interconnected so as to affect said efferent 
signal by gain control signal from w c cell and to affect 
efferent signal by gate signal from u*cell to w,cell; and 
the feature-extracting cell u s is affected by feedback 
signal x from a maximum detector in the highest stage, 



said efferent signals are processed by a plurality of 3 „ md the int tive feature .i„ d icating cell w, is affected 



integrative feature-indicating cells w c and feature- 
indicating cell w, alternatively and divergingly; 
cell layers in each stage, except input-cell layer U 0 of 
the lowest stage and the cell layer in the highest 
stage, having converging afferent connections 35 
from feature-extracting cells u* to feature-integrat- 
ing cells u c , and diverging efferent connections 
from an integrative feature-indicating cell w c to 
feature-indicating cell w^; 
a threshold controlling cell Wcx for receiving signals 40 
from the cell w c via unmodifiable excitatory con- 
nections and receiving converging signals form the 
cells u s in the same stage and controlling the selec- 
tivity for extracting particular features by said fea- 
ture-extracting cells u,; 45 
wherein each feature-extracting cell is connected 
to send an unmodifiable excitatory signal or gate 
signal to each feature-indicating cell w* ( and each 
integrative feature-indicating cell w c is connected 
to send a gain control or heterosynaptic facilitation 50 
signal to a feature-integrating cell u c , and the trans- 
mission of efferent signals is gated by the afferent 
signals in the cell w Jt and simultaneously, the trans- 
mission of afferent signals is facilitated by the pres- 
ence of the efferent signals in the cell w c ; 55 
whereby a pattern corresponding to the cell which is 
yielding the largest output among the feature-inte- 
grating cells u c of the highest stage is interpreted as 
recognized; and the output of the cell layer w co of 
integrative feature-indicating cells of the lowest 60 
stage being interpreted as the output of associative 
recall or as the result of segmentation of a signal 
pattern which is now being recognized. 
2. A hierarchical information processing system for 
processing information in parallel over a plurality of 65 
cell layers as defined in claim 1, further comprising 
feedback means for transmitting a signal x derived from 
a maximum detector when the maximum output is zero 



by an unmodifiable inhibitory signal from the feature- 
indicating inhibitory cell w^ activated by feature- 
indicating cell Wj. 

6. A hierarchical information processing system for 
processing information in parallel over a plurality of 
cell layers comprising; 
a plurality of afferent signal paths extending from 
lower order cell-layers to higher order cell-layers 
for processing information signals, wherein said 
afferent signals are processed by a plurality of fea- 
ture-extracting cells u^and feature-integrating cells 
u c alternatively and convergingly; 
a plurality of efferent information paths extending 
from higher order cell-layers to lower order cell- 
layers for transmitting an efferent signal, wherein 
said efferent signals are processed by a plurality of 
integrative feature-indicating cells w c and feature- 
. indicating cells w, alternatively and divergingly; 
cells layers in each stage, except input-cell layer U 0 of 
the lowest stage and the cell layer in the highest 
stage, having converging afferent connections 
from feature-extracting cells u s to feature-integrat- 
ing cells u Cp and diverging efferent connections 
from an integrative feature-indicating cell w c to a 
feature-indicating cell w f ; 
an inhibiting cell u.^ convergingly receiving a pattern 
signal from the said feature-integrating cell \ic of 
the preceding stage and sending output signals via 
modifiable inhibitory connections to the said fea- 
ture-extracting cell u 5 ; 

a threshold controlling cell w„ receiving signals 
from the cell w c via unmodifiable excitatory 
connections and receiving converging signals 
from the cells Ujin the same stage and controlling 
the selectivity for extracting particular features 
by said feature-extracting cell v s ; 

a feature-indicating inhibitory cell w JV connected to 
the feature-indicating cell w, and connected di- 
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vergingly and unmodifiably to the integrated 
feature-indicating cell w c in the lower stage; 

wherein each feature-extracting cell Ujis connected 
to send an unmodifiable excitatory signal or gate 
signal to each feature-indicating sell w* and each 
integrative feature-indicating cell w c is con- 
nected to send a gain control or heterosynaptic 
facilitation signal to a feature-integrating cell u 0 
and the transmission of efferent signals is gated 
by the afferent signals in the cell Wj, and simulta- 
neously, the transmission of afferent signals is 
facilitated by the presence of the efferent signals 
in the cell w c ; 

whereby a pattern corresponding to the cell which 
is yielding the largest output among the feature- 
integration cells u f of the highest stage is inter- 
preted as recognized; and the output of the cell 
layer w ffl of integrative feature-indicating cells of 
the lowest stage being interpreted as the output 
of associative recall or as the result of segmenta- 
tion of a single pattern which is now being rec- 
ognized. 

7. A hierarchical information processing system as 
defined in claim 6, further comprising feedback means 
for transmitting a signal x derived from a maximum 
detector when the maximum output is zero to all fea- 
ture-extracting cells Uj so as to lessen the inhibition to 
each target cell u y . 

8. A hierarchial information processing system for 
processing information in parallel over a plurality of 30 
cell layers comprising: 

a plurality of afferent signal paths extending from 
lower order cell-layers to higher order cell-layers 
for processing information signals, wherein said 
afferent signals are processed by a plurality of fea- 35 
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ture-extracting analog cells u,and feature-integrat- 
ing analog cells u c alternatively and convergingly; 
a plurality of efferent information paths extending 
from higher order cell-layers to lower order cell- 
layers for transmitting an efferent signal, wherein 
said efferent signals are processed by a plurality of 
integrative feature-indicating analog cells w c and 
feature-indicating analog cell w, alternatively and 
divergingly; 

cell layers in each stage, except input-cell layer U tf of 
the lowest stage and the cell layer in the highest 
stage, having converging afferent connections 
from feature-extracting cells u, to feature-integrat- 
ing cells Uc, and diverging efferent connections 
from an integrative feature-indicating cell w c to 
feature-indicating cell w x ; 

wherein each feature-extracting cell u s is connected 
to send an unmodifiable excitatory signal or gate 
signal to each feature-indicating cell w* and each 
integrative feature-indicating cell w c is connected 
to send a gain control or heterosynaptic facilitation 
signal to a feature-integrating cell u ft and the trans- 
mission of efferent signals is gated by the afferent 
signals in the cell w A and simultaneously, the trans- 
mission of afferent signals is facilitated by the pres- 
ence of the efferent signals in the cell w^ 

whereby a pattern corresponding to the cell which is 
yielding the largest output among the feature-inte- 
grating cells u c of the highest stage is interpreted as 
recognized; and the output of the cell layer w„ of 
integrative feature-indicating cells of the lowest 
stage being interpreted as the output of associative 
recall or as the result of segmentation of a single 
pattern which is now being recognized. 
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